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(57) Abstract 

An electronic device (10) receives inputs using images of input devices (34a) and scanning sensors (14, 16, 22) that detect user 
interaction with those images. For example, an image of a mouse (346) may be projected and the user's interaction with that image may 
be scanned using digital video cameras (14, 16, 22). These user movement images may then be analyzed to determine, for example, the 
direction, extent and rate of movement of the user to provide pointer position command signals. Similarly the user's movement relative to 
a virtual keyboard image (34b) may be analyzed and converted into key actuation signals for responding to the actuated virtual keys. 
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Input Device Using Scanning Sensors 

Background 

This invention relates generally to input devices for electronic devices including 
computer systems and particularly to such devices using video control inputs. 

5 A variety of pointing devices have been used in computer systems to control the 

position of a pointer or cursor on a display. A pointer or cursor is an image that may be 
moved across the screen to select a particular option provided in the form of a physical 
depiction on a user interface. In this way, the user may interact with a video display to 
provide focus to particular tasks and activities offered through the user interface. 

10 Known pointing devices include the mouse, track ball and the pointing stick. All of 

these devices generally involve an electromechanical interface which is prone to failure. 
When the interface failsTit may resulFin an incorrect signal"wMch may COTSFmUveffienraf- 
the pointing device to be incorrectly reflected in the pointer position of the display. 



Another well known pointing device, the touch pad pointing device, allows the user to 
15 touch a contact surface. Movements of the user's finger on the contact surface are tracked as 
corresponding movements of the pointer on the computer display. Normally, touch pads have 
capacitive sensors which detect finger position and movement and translate that movement 
into position control signals. 

The touch pad pointing devices are subject to being fouled by dirt or moisture, 
20 resulting in incorrect signals as described above. In addition the touch pad devices provide a 
limited degree of movement, making it difficult to learn to control the pointer using the 
limited movements available through the pointing devices. 

A number of computer systems use keyboards for inputting text information to the 
computer. The keyboard is generally connected to the computer by a cable. Like the 
25 pointing device, the keyboard involves an electromechanical interface which is prone to 

failure. These failures may result in the inability of a particular key to activate a particular 
input. In any case, inputs which are provided to the keyboard may be inaccurately reflected 
in the text which appears on the display screen. 

Thus, there is a need for an improved input device which allows the user to input 
30 information in ways which may be less prone to failure. 

A number of electronic devices, including computer systems, include relatively bulky 
input devices such as keyboards. For example, in many portable computers such as laptop or 
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notebook computers, a folding housing is used in which the display is essentially one portion 
of the overall housine and the keyboard is the other portion. Thus, a considerable portion of 
the overall bulk of the device is taken up by the keyboard. Similarly in some personal digital 
assistants(PDAs) or palm-sized computers, the keyboard or keypad input dev 1C e takes up a 
5 considerable portion of the overall bulk of the device. 

There is a demand for ever smaller computing devices. Portable computers such as 
laptop or notebook computers are progressively decreasing in size. Similarly there is 
increasing interest in palm-sized or PDA-type computer systems. 

Thus, there is a continuing demand for new ways to reduce the size of electronic 
10 devices including computer systems. ^ ^ 



"Summary" 



In accordance with one embodiment, a method for inputting data to an electronic 
dev 1C e includes projecting an image on a surface. User movements are detected relative to 
1 5 the projected image. 

Rrief Description Of The Drawings 

Figure 1 is a side elevational view of one embodiment in accordance with the 

invention; 

20 Figure 2 is a front elevational view of the embodiment shown in Figure 1 ; 

Figure 3 is a top plan view of the images which may be projected by the device shown 

in Figure 1 ; 

Figure 4 is a block diagram of one embodiment in accordance with the invention for 
implementing the device shown in Figure 1 ; 
>> 5 Figure 5 is a flow diagram of the process of forming a device which can provide the 

features of the device shown in Figure 1 in one embodiment in accordance with the invention; 

Figure 6 is a flow diagram for one software implementation of the embodiment shown 
in Figure 1 ; and 

Figures 7 and 8 are flow diagrams for calibration software in one embodiment in 
30 accordance with the invention. 
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Detailed Description 

An electronic device which may be implemented as the computer system 1 0 5 shown in 
Figure L may include a housing 12 which includes a display 18. Unlike conventional 
computer systems, the system 1 0 may not have a keyboard or a pointing device. Scanning 

5 sensors 1 4 and 1 6 may be provided on extended arms 26 and 30 to enable scanning of the 
region m front of the housing 12. The scanning sensors 14 and 1 6 may, for example, be 
digital cameras that produce streaming video which is conducted through cables contained in 
the arms 26 and 30 to the housing 12. An additional scanning sensor 22 may be provided on 
the front face of the housing 12 beneath the display 1 8, as shown in Figure 2. 

10 Also situated on the housing 12 is a projector 20 that projects an image onto a surface, 

such as a desktop, situated in tront ot the Housing u. The projector 20 may for example, be a 
li quid crystal display (L CD) projector, a laser p roj ector or a light engine projector such as a 
digital light processing (DLP) light engine from Texas Instruments, Inc. The region where 
the image is projected may be the same region which is scanned by the scanning sensors 14, 

15" t6-and-22— Each-scanner may-be-aligned transversely-to-each-of-the-other-scanners for-better- 

three dimensional imaging. The housing 12 may be supported at an angled orientation by the 
stand 24. 

In one embodiment in accordance with the invention, one or more input devices, 
shown in Figure 3, such as a virtual keyboard 34a and mouse 34b may be projected onto a 

20 surface in front of the housing 12. However, the invention is not limited to this illustrated 
embodiment. When the user positions his or her hands over the image as the user would in 
using a physical input device, the scanning sensors may capture the movement of the user's 
hands and convert this movement into input command signals. For example, movements of 
the user's hand over the surface upon which the mouse image 34b was projected may be 

25 converted into pointer position signals for moving the position of the pointer on the display 
18. Similarly, typing movements of the user's hands over the surface upon which the 
keyboard image 34a -was projected may cause the corresponding textual input to be entered 
and to appear on the display 18. Thus, an electromechanical input device may be eliminated. 
Alternatively, the scanning sensors may sense the projected light reflected from the 

30 user's hands. The scanning sensors may also detect the combination of the reflected, 

projected light and the user's hands. Also, the projected images may move so as to "follow" 
the detected movement of the user's hands. 
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Turmng now to Figure 4, an example of a computer system 35 for implementing one 
embodiment in accordance with the invention may include a processor 36 coupled to a host 
bus 38. The illustrated computer system 35 includes the 440BX chip set with a Pentium®!! 
processor available from Intel Corporation. The host bus 38 is coupled to a host bridge 40 
winch in turn couples a mam memory 42. The host bridge 40 also couples an accelerated 
graphic port (AGP) bus 48 (see Accelerated Graphics Port Interface Specification, Revision 
1.0. published on July 31. 1996, by Intel Corporation of Santa Clara, California) which 
couples a graphic dev 1C e 50. The graphics dev 1C e 50 may receive inputs from the scanning 
sensors 14. 16 and 22 and may display the resulting inputs on a display 1 8 which is also 
coupled to the graphics device 50. The graphics device 50 may also have a video basic 
input/output system (BIOS) 54 and a graphics local memory 56. 



;uvpwi ^ j ^-- r - — \ / — 

^ h ^n H 4ge^Q4s^se-^ 



15 



20 



25 



30 



46 for receiving peripheral devices. The bus 44 is also coupled to a bus-to-bus bridge 58 
which is in tum coupled to an additional bus 70. The bus 70 includes a plurality of slots 72 

TSFrecSvi^^ 5 *™ y alS ° 

couple to a plurality of ports 60 and 66. The port 60 may be coupled, for example, to a hard 
disk drive 62 which stores one or more programs 64, 106 and 122 as indicated. The port 66 

mav be coupled to the projector 20. 

While one example of a computer system is shown in Figure 4, those skilled in the art 
will appreciate that a number of other systems may be utilized to achieve the objectives set 
forth herein. Therefore it should be understood that the system illustrated is only an example. 

Turning now to Figure 5, one embodiment of a method for providing the devices 
having the previously described functionality is illustrated in accordance with the invention. 
Initially a system is provided with video capability, as indicated in block 76. For example, a 
computer system may be provided with the ability to process a plurality of streaming video 
inputs. The system may also be equipped with a plurality of digital cameras. A projector is 
enabled to project an image of an input device on a surface, as indicated in block 78. The 
ability to provide video analysis of the resulting movements of the user's hands, for example, 
may also be provided as indicated in block 80. In accordance with one embodiment, the video 
analysis may be a conventional pattern recognition software which recognizes the shape of 
the user's hand and corresponding movements thereof. Next, the ability to convert the user's 
movements in association with the projected images to a pointer position control signal may 
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be provided, as indicated in block 82. Finally, the ability to convert the movements of the 
user into text inputs may be provided, as indicated in block 84. In addition, an audible 
confirmation, such as a key clicking sound may be produced when a keystroke is recognized. 
Turning now to Figure 6, software 64 may be stored on the hard disk drive 62 in one 
5 embodiment according to the invention. Initially the software causes the image of one or 

more input devices to be projected onto a surface. For example, the image of a keyboard and 
a mouse may be projected, as indicated in block 86. Thereafter streaming video may be 
received (block 87) and a particular video frame may be selected, as indicated at block 88. 
The frame may be selected randomly, or at regular intervals, for example, using a selection 
1 0 technique. 

In addition, a scene change detector may be used to determine when Ihere is a ; 

signific ant change in scene. When a significant scene change occurs, a frame may be 

captured for subsequent analysis. Scene change detectors are known in the art and may 
involve an analysis of the pattern of intensity signals determined by the pixels that make up 

—f5— — mTmagmgtnrayn^ 

provide output signals. If an output signal is above a certain intensity level, it may be 
considered white and if it is below a certain level it may be considered to be black. A 
significant change in the number of white versus black output signals can be recognized as a 
scene change initiating the selection of one frame from the streaming video. 
20 While the present invention has been described with respect to the use of a single 

frame for this analysis, a continuing analysis can be done by successively or continually 
analyzing successive frames. Oversampling may be used to insure that movements are 
detected. 

In the embodiment illustrated in Figure 6, once a frame is selected, as indicated' in 
25 block 88, conventional pattern recognition software may be utilized to determine if a 

particular image pattern is represented by the digital video data stream, as indicated in block 
92. This software may recognize a particular preprogrammed image such as the position of 
the user's fingers on a virtual keyboard or the position of the user's hand on top of a virtual 
mouse. This software may recognize the image by comparing it to a previously recorded 
30 depiction or the software may be trained to learn a particular image through cooperation of 
the computer owner or user, using learning software. 



): <WO 0021024A1 i > 



10 



PCT/US99/19598 

WO 00/21024 

-6- 

. At diamond 94, the software checks to determine whether the selected frame contains 
a pattern matching a predetermined pattern which corresponds to a particular input command. 
If not flow returns to select another frame. 

If a pattern match is detected, the first and a subsequent frame are compared as 
indicated in block 96. Next, a check detemunes whether the comparison indicates a change 
in position (diamond 98). If so, the rate and direction of movement of the unage may be 
determined (block 100). The rate and/or direction of movement can be used to provide 
corresponding command inputs. For example, in connection with the movement of the user's 
hand in association with the mouse nnage, the direction and rate of movement can be 

reflected in corresponding movements of the pointer on the display 18. For example, a 

^^^*Z^X* the three dimensional position ox an on screen 



«„ move the pointer on the display 18 aeeording to the direcnon or rate of movement of the 
user's hand (block 104). The flow cycles back to select a new frame (a, block 88) to star, the 

"lT "compSison again. 

Alternatively, a distinct object may be associated with the user so that the object may 
be easily recognized. For example, a distinctly colored nng may be used which can be 
readily detected. Movement of the ring may be detected as a pomter position command. 

Software 1 06 for enabling the system to calibrate a focus point for pointer position 
20 commands is illustrated by Figure 7. The software 106 may be utilized during a calibration 
period in order to program the system to recognize the desired inputs. Referring to Figure 7, 
initially streaming video is received as indicated in block 108. A frame of that streaming 
video is captured, as illustrated in block 110. 

Next the system is prompted for movement in a defined direction as shown in block 
25 112 For example, an on screen display may guide the user through the learning process. For 
example, the on screen display may first direct the user to show a left cursor position 
command. The sequence then cycles through each of the cursor command directions. Once 
the system knows what cursor command is being programmed, it may capture a new frame of 
video as indicated in block 114. The new frame and the pnor frame are compared to 
30 determine a difference that amounts to a given cursor command (block 1 1 6). A check at 
diamond 1 1 8 detemunes whether a focus point has been identified. In other words, the 
system must be able to differentiate the two frames sufficiently to be able to thereafter 
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recognize those differences as a particular pointer position command. If not, the process is 
tried again. Otherwise, the system saves off the focus point data of the identified focus point, 
as indicated in block 120. 

Referring to Figure 8, the software 122 enables a calibration of speed and direction of 
5 pointer position commands. The focus point data determined by the software 106 is loaded at 
block 123. Streaming video is received as indicated at block 124 and a frame is captured as 
shown in block 126. A focus point is identified such as the user's finger as indicated in block 
128 and a movement prompt is provided (block 130). The prompt may be a message block 
on the screen' asking the user to implement what the user wants to use as a desired movement. 

1 0 A new (subsequent) frame is captured in block 132 and compared to the frame captured in 

block 126, as illustrated by block 134. 

The focus point movement is identified (block 136) and a direction and rate of 

movement are calculated as indicated in block 138. A correlation factor for x s y and z 
directions is used to calculate the corresponding rate and direction of movement of the screen 

15 pointer-(Woek-l-40)^^e-sereefi 

then may adjust the amount of pointer movement for a given movement of the user's finger, 
for example through inputs provided by a prompt screen, as indicated in block 144. In 
addition, the user may provide alternative inputs which could be received as a particular 
- pointer position command. The user is then prompted to indicate whether the user is done in 
20 diamond 146. If not, the flow cycles and new commands may be learned as described above. 
Otherwise the flow is terminated and the. information is saved (block 148). 

The number of cameras that may be used may be subject to considerable variation. 
More cameras may result in better three dimensional imaging. This may allow movements in 
three dimensions to be recognized as input command signals. In addition, camera 
25 redundancy may be used to eliminated errors. 

While the present invention has been illustrated with respect to separate cameras 
mounted on a computer, the cameras could be integrated into the housing of the computer 
system itself. In addition, while the invention has been illustrated in connection with cameras 
sensitive to light in a visible spectrum, infrared detecting cameras could be used as well 
30 By using streaming video to control input information, an electromechanical 

connection (which may be prone to errors or failure) may be eliminated in some 
embodiments. In addition, the accuracy of the inputs may be improved in some 
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embodiments, for example, as compared to touch pad pointing devices, because the 
possibility of incorrect actuation due to dirt or moisture may be eliminated. 

Other input devices may be handled in the same fashion. For example, a telephone 
may be replaced with the projected image of a telephone. A computer microphone may be 
5 activated when the handset is grasped and lifted as recognized by the scanning sensor system. 
Dialing could be handled in the same fashion described previously in connection With 
actuation of the keyboard projected image. 

The projected image may be altered to correspond to the portion of the input device 
the user sees with his or her hands positioned over a normal input device. This altered image 
10 may be produced when the user's hands are detected as bei ng in position over the surface that 
received the projected image. Moreover, the input device image may itself be moQined 10 



15" 



corresponds to key depression, the projected image may be altered to suggest that the virtual 
key has moved. 

WHflFthe^e^rinvention has been-described-with-respect to ^limited number of 

embodiments, those skilled in the art will appreciate numerous modifications and variations 
therefrom. It is intended that the appended claims cover all such modifications and variations 
as fall within the true spirit and scope of the present invention. 
What is claimed is: 
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1 1 . A method for inputting data to an electronic device comprising: 

2 projecting an image on a surface; and 

3 detecting user movements relative to said projected image. 

1 2. The method of claim 1 further including capturing a digital video image of the 

2 user in association with the projected image and inputting a representation of said image to 

3 the electronic device. 

1 3. The method of claim 2 further including analyzing said digital video image 

2 including using pattern recognition software. 



4 4, T^m^d-of^to ajdirection-Qfl movement 

2 of an object depicted in said digital video image. 



3T - "~TTK^effi5ar^^^ 

surface an image of a mouse and detecting user movements relative to said mouse image. 

6. The method of claim 1 wherein projecting an image includes projecting the 
image of a keyboard and detecting user movements relative to said keyboard image. 

7. The method of claim .1 wherein projecting an image includes projecting an 
image of an input device. 

8. An article comprising a medium for storing instructions that cause a computer 

to: 

project an image on a surface; and 

detect user movements relative to said image. 

9. The article of claim 8 further including instructions that cause a computer to 
analyze a digital video stream using pattern recognition software. 
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1 10. The article of claim 9 further including instructions that cause a computer to 

2 determine a direction of movement of an object depicted in said digital video stream. 

1 1 1 The article of claim 8 including instructions that cause a computer to project 

2 an image of a mouse on a surface and detecting user movements relative to said mouse image. 

1 1 2. The article of claim 8 including instructions that cause a computer to project 

2 the image of a keyboard and detecting user movements relative to said keyboard image. 



1 



13. The article of claim 8 including instructions that cause a computer to project 



2 the image of an input device. 



1 14. A system comprising: 

2 a processor; 

3 Tsensor coupled to said processor; 

4 a projector coupled to said processor; and 

5 a memory associated with said processor, said memory storing a computer 

6 program that causes said processor to: 

7 project an image of an input device using said projector; and 

g use said sensor to detect user movement relative to said image of an 

9 input device. 

1 15. The system of claim 14 including a plurality of digital video cameras. 

1 1 6. The system of claim 1 4 wherein said program controls the position of a pointer 

2 in response to movements of a user. 

1 17. The system of claim 14 wherein said system is a computer system. 

1 1 8. The system of claim 1 4 wherein said memory includes pattern recognition 

2 software. 
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1 19. The system of claim 1 4 wherein said proj ector proj ects an image of a pointing 

2 device. 

1 20. The system of claim 1 4 wherein said projector projects an image of a 

2 keyboard. 
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